Early and Cost-effective Software Fault Detection Measurement and Implementation in an Industrial Setting

Avoidable rework consumes a large part of development projects, i.e. 20-80 percent depending on the maturity of the organization and the complexity of the products. High amounts of avoidable rework commonly occur when having many faults left to correct in late stages of a project. In fact, research studies indicate that the cost of rework could be decreased by up to 50 percent by finding more faults earlier. Therefore, the interest from industry to improve this area is large. It might appear easy to reduce the amount of rework just by putting more focus on early verification activities, e.g. reviews. However, activities such as reviews and testing are good at catching different types of faults at different stages in the development cycle. Further, some system characteristics such as system capacity and backward compatibility might not be feasible to verify early through for example reviews or unit tests. Therefore, the objective should not just be to find and remove all faults as early as possible. Instead, the cost-effectiveness of different techniques in relation to different types of faults should be in focus. A department at Ericsson AB was interested in approaches for assessing and improving early and cost-effective fault detection. In particular, there was a need to quantify the value of suggested improvements. Based on this objective, research was during a few years conducted in the industrial environment. The conducted research resulted in this thesis, which determines how to quantify unnecessary rework costs and determines which phases and activities to focus improvement work on in order to achieve earlier and more cost-effective fault detection. The thesis describes and evaluates measurement methods that make organizations strive towards finding the right faults in the right phase. The developed methods were also used for evaluating the impact a framework for component-level test automation and test-driven development had on development efficiency and quality. Further, the thesis demonstrates how the implementation of such improvements can be continuously monitored to obtain feedback during ongoing projects. Finally, recommendations on how to define and implement measurements, and how to interpret obtained measurement data are provided, e.g. presented as considerations, lessons learned, and success factors. The thesis concluded that existing approaches for assessing and improving the degree of early and cost-effective software fault detection are not satisfactory since they can cause counter-productive behavior. An approach that more adequately considers the cost-efficiency aspects of software fault detection is required. Additionally, experiences from different products and organizations led to the conclusion that a combination of measurements is commonly necessary to accurately identify and prioritize improvements.